Interactive window

ABSTRACT

A method performed by a system. The method includes automatically generating an interactive window on at least a portion of a display, the interactive window having a first size corresponding to a first pattern on the display. The method further includes automatically expanding the interactive window to a second size in response to a second pattern on the display to enclose the first pattern and the second pattern in the interactive window.

BACKGROUND

Effective communication between different parties is an important partof today's world. With the increased availability of high-speed networkconnectivity, video conferencing conducted over networks betweenparticipants in different locations has become very popular. Remotecollaboration and videoconferencing systems enable remotely locatedusers at several different sites to simultaneously collaborate with oneanother via interactive video and audio transmissions. A user at onelocation can see and interact with a user at other locations inreal-time and without noticeable delay. At times during interactivecollaborations, the systems may display content on the screens of thelocal and/or remote users. In addition, the systems offer variouscollaborative tools that help the users to communicate and shareinformation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of two users video-conferencing through a display.

FIG. 2 is a schematic diagram illustrating an example of adisplay-camera system that generates interactive windows.

FIG. 3 is a schematic diagram illustrating an example of a computingdevice of the display-camera system of FIG. 2.

FIG. 4 is a flow chart illustrating an example of a method forautomatically generating and expanding an interactive window.

FIGS. 5A-5C are schematic illustrations showing examples of generatingand expanding an interactive window.

FIGS. 6 and 7 are flow charts illustrating examples of alternativemethods for generating and modifying an interactive window.

FIGS. 8A-B are schematic illustrations showing examples of generatingand modifying an interactive window that includes media content.

DETAILED DESCRIPTION

In the following detailed description, reference is made to theaccompanying drawings, which form a part hereof, and in which is shownby way of illustration specific examples in which the disclosed subjectmatter may be practiced. It is to be understood that other examples maybe utilized and structural or logical changes may be made withoutdeparting from the scope of the present disclosure. The followingdetailed description, therefore, is not to be taken in a limiting sense,and the scope of the present disclosure is defined by the appendedclaims. Also, it is to be understood that the phraseology andterminology used herein is for the purpose of description and should notbe regarded as limiting. The use of “including,” “comprising” or“having” and variations thereof herein is meant to encompass the itemslisted thereafter and equivalents thereof as well as additional items.It should also be noted that a plurality of hardware and software baseddevices, as well as a plurality of different structural components maybe used to implement the disclosed methods and systems.

As used herein, the term display-camera system refers to interactivecollaboration and videoconferencing systems that share digital audio orvisual media between remote users. The terms local site and remote siteare descriptive terms that define a physical separation between thedescribed systems, persons, or objects and other systems, persons, orobjects. The physical separation may be any suitable distance betweenlocations such as a short distance within the same room or betweenadjacent rooms of a building or a long distance between differentcountries or continents. The term local user refers to a person whoviews a local display-camera system, and the term remote user refers toa person who views a remote display-camera system.

As used herein, the term automatically refers to acting or operating ina manner essentially independent of external human influence or control.A system or an element of a system may have the capability to start,operate, perform functions, etc., independently of specific requests orregulations.

As used herein, the term content refers to visual and/or audiorepresentation and information (e.g., text, image, video, etc.) that isreproduced for a user using any suitable computing or rendering (e.g.,video or audio) output device for the display-camera systems describedherein. The content may be displayed on displays of the display-camerasystems. The content can also be created by a user through a use of aninterface, such as a touch or contact-sensitive interface (e.g., adisplay), or through an input mechanism such as a pointer device (e.g.,stylus, electronic pen, mouse, etc.) or a keypad. The term sharedcontent refers to content that is displayed on both a localdisplay-camera system and a remote display-camera system so the samecontent is simultaneously visible by a local user and a remote user.

The display-camera system can also allow interactive windows (i.e.,added content) to be displayed on a display of the display-camerasystem. An interactive window (e.g., a virtual whiteboard) may define aspecific area that includes a distinctive background and may include anysuitable visual and/or audio information produced or displayed by anapplication or other suitable program. In one example, the interactivewindow allows local and remote users to generate content visible to bothlocal and remote users. For example, a user of the display-camera systemcan view a whiteboard displayed on the display and can interact (e.g.,draw, mark, etc.) on the whiteboard by using a pointer device or afinger (i.e., when the display has a contact-sensitive surface). As usedherein, the terms interactive window and whiteboard are usedinterchangeably such that interactive window may also be referred towhiteboard and whiteboard may also be referred to as interactive window.

The terms overlaps and overlapping, when used with reference to thecontent displayed on the display-camera system, refer to at leastportions of content (e.g., a video) and/or interactive window (e.g., awhiteboard) that are displayed on a common portion on the display. Theterm semi-transparent, when used with reference to interactive window,indicates that the interactive window or a portion of the interactivewindow is at least partially transparent so another content, includingthe displayed background image that overlaps with the interactivewindow, is at least partially visible to a user through the interactivewindow if portions of these contents overlap.

As described herein, a display-camera system is configured to displayremote content, to capture local content, and to generate and display atleast one interactive window (i.e., a whiteboard) at the same time. Forexample, the display-camera system receives remote content from a remotesystem (e.g., a remote display-camera system located in a remote site)that may include captured images of the remote site or a remote user anddisplays the remote content to a local user using an image renderingdevice (e.g., a projector) or a display. The display-camera system mayalso capture the local site and the local users through the displayusing a camera and transmits the that that to the remote system.Further, the display-camera system allows interactive windows (e.g.,electronic whiteboards) to be displayed on a display of thedisplay-camera system in response to patterns (e.g., virtual or physicalmovements or strokes by a user) on the display.

Users of the display-camera system frequently take advantage of theability of the system to generate and display interactive windows. Forexample, many users routinely utilize a whiteboard as an interactive andcollaborative tool for communication. A whiteboard allows the users toenter, display, share information with other users, manipulate thecontents of that whiteboard as a whole (e.g. move or scale thewhiteboard), and to save that shared information.

Operating the display-camera system in order to adjust, resize, and saveinformation entered to a whiteboard or any other interactive windowdisplayed by the system can present various issues to a user. Forexample, a user may want to save information entered on the whiteboardso that information can be distributed or shared at a later time. If anexisting whiteboard covers the entire display of the display-camerasystem, a user may end up saving all the data entered on the whiteboardalthough he or she may only need to save a fraction of that data. Thismay require users to store and manipulate much more data than they wish.Therefore, users of such systems are always burdened with an additionaltask when using a whiteboard: they must either select a specific size ofthe whiteboard before start using it (i.e., in order to minimize thedata that is saved), or they must select a specific portion of thewhiteboard to be saved after the whiteboard is created. In any event,users must perform specific functions in order to save and/or manipulatedata that is covered or inked on the whiteboard.

This description is directed to methods, systems, and non-transitorymachine-readable storage medium that automatically create an interactivewindow on a display of a system and automatically adjust the size of theinteractive window based on the virtual or physical patterns by a user.The proposed methods allow users to concentrate on generating andmanipulating the data included on the interactive window without concernfor creating, sizing, and saving the window. Therefore, users can devotetheir full attention to the substance of the data on the interactivewindow and the creation and sizing of the window is performedautomatically based on the user's actions.

In one example, a proposed method involves automatically generating aninteractive window on at least a portion of the display, where theinteractive window has a first size corresponding to a first pattern onthe display. The interactive window of the first size defines a defaultmargin extending outside of a bounding area of the first pattern. Theproposed method also involves automatically expanding the interactivewindow to a second size in response to a second pattern on the displayto enclose the first pattern and the second pattern. The second patternis within a predetermined time interval from the first pattern and thesecond pattern is at least outside of the bounding area of the firstpattern to automatically expand the interactive window to the secondsize.

FIG. 1 shows an example of two video-conferencing users interactingthrough a display 102. The display 102 enables a local user 104 tointeract with a remotely located user 106. In the illustrated example, acamera 108 and an image rendering device 202 (e.g., a projector) arepositioned behind the display 102. An interactive window or whiteboard110 is displayed on the display 102. The camera 108 may be positioned atapproximately eye level to the local user 104, to capture images of thelocal user 104 through the display 102. The camera 108 can be positionedso that its viewpoint is roughly equivalent to that of the remote user106. In order to capture gestures made by the local user 104, the camera108 can also be positioned so that the camera's field of viewencompasses approximately the entire display 102. In one example, thecamera 108 and the projector 202 are connected to a computing device 204having a processor 302, a memory 310, and an interactive window module318 that controls the operation of the whiteboard 110 according to theproposed method.

FIG. 2 is a schematic illustration of an example display-camera system200 that generates interactive windows 110 (shown in FIG. 1). It is tobe understood that the system 200 is illustrated only as an example andthe methods described below can relate to any visual collaborationsystem (e.g., a system including at least two end-points thatcommunicate over a network, where each end-point has at least a displayand a computing device) where users can display shared content and useinteractive windows. The illustrated system 200 includes the display102, the camera 108, the projector 202, and the computing device 204including the processor 302, the memory 310, and the interactive windowmodule 318. The camera 108 and projector 202 are connected to thecomputing device 204, and the camera 108 and projector 202 arepositioned to face the rear surface 206 of the display 102. In oneexample, the computing device 204 is in communication with a remotedisplay-camera system 209 via a communication link 210. In one example,the camera 108 and projector 202 may be configured such that an imageprojected on the display 102 is not visible to the camera 108, so thatthe camera 108 only captures a view of the local user 104, but not ofthe contents displayed on the screen 102.

The display-camera system 200 receives remote content 220 from theremote display-camera system 209 using the link 210 where the remotecontent 220 may include captured images of a remote site including theremote user 106. In one example, the display-camera system 200 formsdisplay content 230 from the remote content 220 and added content 225(e.g., interactive windows 110, media content, etc.), if any, from thelocal and/or remote users and provides the display content 230 to thedisplay 102. The added content 225 may be any suitable visual and/oraudio information produced or displayed by an application or othersuitable program. For example, the added content 225 can include aninteractive window that allows local and/or remote users to generatecontent visible to both the local and the remote users. Further, theadded content 225 can include content added by a user of the system 200or content received from various external devices connected to thesystem 200. When the added content 225 is in the form of an interactivewindow 110, the interactive window may be displayed in conjunction withthe remote content 220 in some examples. The process for generating andadjusting an interactive window performed by the interactive windowmodule 318 is described in detail below. In another example, the display208 of the system 200 displays unshared content (not show) that is notdisplayed on the remote system 209 (e.g., alert boxes, save dialogs,private data such as local file directories, etc.).

In additional examples, the added content 225 can include media inputfrom, for example, a device or computer. The media input can include amedia component that is shared between the system 200 and the remotesystem 209. The media input can include, for example, a document, animage, a video, a slide show or presentation, or other media content.Further, the added content 225 can include various interaction inputsfrom local user or the remote user (e.g., generation of electronic ink,etc.). In one example, the added content 225 can also be shared contentthat is displayed on both the local display-camera system 200 and thesystem 209 so the same content is simultaneously visible by the localuser 104 and the remote user 106. For example, the computing device 204can receive the added content 225 and can simultaneously send it todisplays in both systems 200 and 209.

The display-camera system 200 displays the display content 230 usingdisplay 102 to produce displayed content 212 that is viewable by thelocal user 104 of display-camera system 200. In one example, thedisplay-camera system 200 projects the display content 230 on thedisplay 102 using the projector 202 to produce the displayed content212. In another example, the display 102 can directly receive thedisplay content 230 from the computing device 204 (e.g., as a sequenceof display images) and can display the display content. Thedisplay-camera system 200 also captures local content 235 at the localsite, including an image of the local user 104 through the display 102using the camera 108 and transmits the local content 235 to remotesystem 209 using the link 210. The type and arrangement of the devicesas shown in FIG. 2 is only shown one example, and many other types andarrangements of devices may be used in other examples.

In the example illustrated in FIG. 2, the projector 202 receives thedisplay content 230 from the computing device 204 and projects thecontent to the display 102. For instance, the projector 202 may receivethe display content 230 as a sequence of display images and can projectthe images in non-overlapping time periods. The projector 202 mayrepresent any suitable type and number of projectors that project imageson the display 102. The projector 202 is positioned to project imagesonto the rear surface 206 within this range of angles so that when thelocal user 102 views the front surface 208 of the display 102 the localuser 104 sees the projected images. In some example's (not shown), theprojector 202 may be positioned in front of the display 102 (e.g., nextto or behind the local user 104) so the projector 202 may project imagesonto the front surface 208 of the display 102.

The display 102 displays the content to the local user 104. The display102 may be composed of a relatively low concentration of light diffusingparticles that diffuse light striking the rear surface 206 within arange of angles. In one example, the display 102 is at least partiallytransparent to allow the camera 108 to capture images of the local user104, through display 102. Also, the display 102 may be sufficientlylarge so as to display life-size images of the remote users of remotesystem 209. In some examples, the display 102 may be a screen, a, atransparent liquid crystal display (LCD), an organic light emittingdiode (OLED) display, or any suitable display.

The display 102 may be a touch or contact-sensitive display thatincludes touch sensors, optical sensors, or other contact sensors toallow for user's input via an input mechanism such as a pointer device(e.g., stylus, electronic pen, mouse, etc.) or a user's finger. Thedisplay 102 may also support gesture recognition operations (i.e.,contactless interaction) where the user may interact with thedisplay-camera system 200 or the remote system 209 by gestures in theair that are captured by depth image sensors on the display 102 and areinterpreted via algorithms in the computing device 204. Therefore, usersmay generate interactive or added content (e.g., by using a whiteboard)that can be shared between the system 200 and the remote system 209.

The camera 108 captures local content 235 (e.g., the image of the localuser) in response to control signals from the computing device 204. Forexample, the camera 108 captures local content 235 as a sequence oflocal images that are captured in non-overlapping time periods. Thecamera 108 provides local images 235 to computing device 204 and/or thelink 210 for further processing or for transmission to the remote system209.

In the illustrated example, the camera 108 is positioned behind thedisplay 102 with a distance that is selected so that field of view 214of the camera encompasses all or substantially all of the front side ofdisplay 102. Alternatively, the camera may also be positioned relativeto the display 102 so that field of view 214 is aligned with all or aselected portion of the front side of display 102 to allow for anon-arbitrary mapping between camera 108 and display 102. The camera 108may represent any suitable type and number of cameras that captureimages through display 102. The camera 108 may be or may include anysuitable image sensor, such as a digital charge couple device (CCD)image sensor or complementary metal-oxide-semiconductor (CMOS) imagesensor. The camera 108 may include a processor (not shown), but may alsobe controlled by the processor 302 of the computing device 204. In oneexample, the camera 108 is configured to not capture any light from thedisplay 102 itself, for example by rapidly alternating betweendisplaying and capturing an image, or by displaying images usingwavelengths of light that are blocked by filters (not shown) from thecamera 108.

The communication link 210 includes any suitable type, number, and/orconfiguration of network and/or port devices or connections configuredto allow display-camera system 200 to communicate with the remote system209 and/or other remote systems (not shown). The devices and connectionsof link 210 may operate according to any suitable networking and/or portprotocols to allow information to be transmitted by display-camerasystem 200 to a network, remote system 209 and/or other remote systems(not shown) or received by display-camera system 200 from a network,remote system 209, and/or other remote systems (not shown). For example,connections between the devices shown in FIG. 2 can be made throughlocal area networks (“LANs”), wide area networks (“WANs”) (e.g., aTCP/IP based network, a cellular network, etc.), public switchedtelephone networks (“PSTNs”), neighborhood area networks (“NANs”), homearea networks (“HANs”), personal area networks (“PANs”), Intranets, theInternet, or any other suitable networks.

The remote system 209 represents any suitable display-camera system,display system, and/or processing system located remotely fromdisplay-camera system 20. The remote system 209 captures and/or providesremote content to display-camera system 200 and receives local contentfrom display-camera system 200.

The computing device 204 provides control signals that control theoperation of the display 102, the camera 108, the projector 202, and thelink 210. The computing device 204 receives remote content 220 from theremote system 209 using the link 210. The computing device 204 generatesdisplay content 230 (including interactive windows) to be displayed andprovides the display content 230 to the projector 202 to project it tothe display 102. Further, the computing device 204 controls theoperation of camera 108 to cause local images to be captured andtransmitted to remote system 209 using link 210. As described inadditional detail below, the computing device 204 implements variousmethods for generation and operation of interactive windows in responseto processor 302 executing instructions or modules (e.g., theinteraction window module 318) stored in the memory 310 of the computingdevice.

FIG. 3 shows a schematic representation of the computing device 204. Thecomputing device 204 can be a server, a desktop computer, a laptop, orany other suitable device configured to carry out video and imageprocessing. The computing device 204 can an independent device or can beintegrated in the projector 202 or the camera 108. The computing device204 includes a processor 302 (e.g., a central processing unit, amicroprocessor, a microcontroller, or another suitable programmabledevice); a projector interface 304; a video or camera interface 306;input interfaces 307, a communication interface 308, and a memory 310.Each of these components is operatively coupled to a bus 312. Forexample, the bus 312 can be an EISA, a PCI, a USB, a FireWire, a NuBus,or a PDS. In other examples, the computer 204 includes additional,fewer, or different components for carrying out similar functionalitydescribed herein.

The projector interface 304 enables the computing device 204 tocommunicate with the projector 202 to display the display content 230 onthe screen 102. The camera interface 306 enables the computing device204 to communicate with the camera 108 to capture and process the localcontent 235. The communication interface 308 enables the computingdevice and the system 200 to communicate with a plurality of networksthrough use of the link 210. As described previously, the communicationinterface 308 can be used to receive remote content 220 from the remotesystem 209 and to send local content 230 to the remote system 209 orother remote systems.

The input interfaces 307 can process added content 225 from a user or anexternal device/system. In one example, the input interfaces include 307at least a media interface 307A and an interaction interface 307B. Themedia interface 307A receives media input from a device or computer andcan include, for example, a connector interface, a storage deviceinterface, or a local or wireless communication port which receives themedia input from a computer of the local user. As noted above, the mediainput may include, for example, a document, an image, a video, a slideshow or presentation, or other media content.

The interaction interface 307B can process various interaction inputsfrom users (e.g., whiteboard entries, electronic inks, etc.). Theinteraction interface 307B can include or utilize sensors, such as touchsensitive sensors or optical sensors, which can be provided with, forexample, the surface of the display 102. In one example, the interactioninterface 307B can interact with the display 102 that includes sensors(e.g., contact-sensitive or touch sensors, optical sensors, proximitysensors, etc., not shown) used to detect user proximity, touch, orpen/ink input. The interface 307B can also detect other forms of inputssuch as contactless interaction from a user (e.g., user's gestures abovethe display) by using depth image sensors or any other suitable sensors.In addition, the interaction interface 307B allows the computing device204 to communicate with a mouse or an input device (e.g., a keyboard)which the user can utilize to create added content.

The processor 302 includes a control unit 320 and may be implementedusing any suitable type of processing system where at least oneprocessor is configured to execute computer-readable instructions storedin the memory 310. The memory 310 includes any suitable type, number,and configuration of volatile or non-transitory machine-readable storagemedia configured to store instructions and data. Examples ofmachine-readable storage media in the memory 310 include read-onlymemory (“ROM”), random access memory (“RAM”) (e.g., dynamic RAM[“DRAM”], synchronous DRAM [“SDRAM”], etc.), electrically erasableprogrammable read-only memory (“EEPROM”), flash memory, hard disk, an SDcard, and other suitable magnetic, optical, physical, or electronicmemory devices. The memory 310 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 302.

The memory 310 may also store an operating system 314, such as Mac OS,MS Windows, Unix, or Linux; network applications 316; and an interactivewindow module 318. The operating system 314 can be multi-user,multiprocessing, multitasking, multithreading, and real-time. Theoperating system 314 can also perform basic tasks such as recognizinginput from input devices, such as a keyboard, a keypad, or a mouse;sending output to a projector and a camera; keeping track of files anddirectories on medium 310; controlling peripheral devices, such as diskdrives, printers, image capture device; and managing traffic on the bus312. The network applications 316 includes various components forestablishing and maintaining network connections, such ascomputer-readable instructions for implementing communication protocolsincluding TCP/IP, HTTP, Ethernet, USB, and FireWire.

The interactive window module 318 provides various computer-readableinstruction components for generating an interactive window or awhiteboard 110 and for modifying the size of the whiteboard 110 based onthe user's actions.

The machine-readable storage media are considered to be an article ofmanufacture or part of an article of manufacture. An article ofmanufacture refers to a manufactured component. Software stored on themachine-readable storage media and executed by the processor 302includes, for example, firmware, applications, program data, filters,rules, program modules, and other executable instructions. The controlunit 320 is configured to retrieve from the machine-readable storagemedia and execute, among other things, instructions related to thecontrol processes and methods described herein.

When executed, the instructions cause the control unit 320 to receiveremote content 220 and added content 225 (e.g., interactive windows) andto generate and provide display content 230 to the projector 202 and thedisplay 102. In one example, the displayed content 212 includes imagesfrom the remote user (i.e., remote content) and an interactive window110 (i.e., a whiteboard) generated by the computing device andmanipulated by a user. The instructions also cause control unit 320 togenerate and provide control signals to the camera 108. The instructionsalso cause the control unit to 320 to generate and provide controlsignals to the link 210 to cause remote content 220 to be received fromremote system 209 and/or other remote systems and cause local content235 to be provided to remote system 209.

The following paragraphs describe methods for automatically generatingand modifying interactive windows 110 performed by the processor 302 ofthe computing device 204. The proposed methods assist users in theirinteractive communication by allowing them to concentrate on thesubstance of the communication without a concern for the collaborativetools (i.e., the interactive window). The creation, sizing, and savingof the whiteboard and the generated content is performed automaticallybased on the user's actions.

FIG. 4 illustrates a flow chart showing an example of a method 300 forautomatically generating and expanding an interactive window. The method300 can be executed by the control unit 320 of the processor 302.Various steps described herein with respect to the method 400 arecapable of being executed simultaneously, in parallel, or in an orderthat differs from the illustrated serial manner of execution. The method300 is also capable of being executed using additional or fewer stepsthan are shown in the illustrated example.

The method 300 may be executed in the form of instructions encoded on anon-transitory machine-readable storage medium executable by theprocessor 302. In one example, the instructions for the method 300 arestored in the interaction window module 318.

The method 300 begins in step 350, where the control unit 320automatically generates an interactive window 110 on at least a portionof the display 102, where the interactive window 110 has a first sizecorresponding to a first pattern on the display 102. When a local user104 and the remote user 106 communicate via a display-camera system, thelocal user 104 can see an image of the remote user 106 on the display102 (see FIG. 1). In one example, the image of the remote user 106 onlytakes a portion of the entire display 102. The users 104 and 106 maywish to interactively communicate with each other in real time via toolsthat add content on the display 102. As mentioned above, in one example,the display 102 can be a contact-sensitive display that includes varioussensors to allow for user's input. Alternatively, the display 102 mayalso support contactless interaction via gesture recognition operations.

Therefore, a user can generate or create added or interactive content225 either by directly marking (i.e., inking) on the display 102 or bygesturing in the air above the display 102. When a user adds suchcontent on the display 102, the user creates a specific pattern on thedisplay 102 based on his or her input. The pattern may be a virtualpattern generated by user's gestures or a physical pattern generated bythe user's direct strokes or inks on the display 102 (e.g., by using apointer device or a finger). Examples of patterns 111 and 111A are shownin FIGS. 5A-5C

FIGS. 5A-5C are schematic illustrations showing examples of generatingand expanding an interactive window 110. As shown in FIG. 5A, a user(e.g., the local user 104) views the remote user 104 and background (notshown) corresponding to the location of the remote user 106 on thedisplay 102. When a user adds content on the display 102 in the form ofthe pattern 111, the control unit 320 automatically generates aninteractive window or a whiteboard 110 on at least a portion of adisplay 102. In one example, the control unit 320 implements a “pendown” function to automatically generate the interactive window 110 assoon as a user inks or gestures on the display 102. In another example,the unit 320 implements a “pen up” function to automatically generatethe interactive window 110 as soon as a user completes a pattern. Insome examples, the interactive window 110 represents shared content thatis displayed on both the local display-camera system 200 and the system209. Therefore, the interactive window 110 is simultaneously visible andeditable by the local user 104 and the remote user 106.

In one example, the interactive window 110 has a first sizecorresponding to the first (i.e., initial) pattern 111 created by theuser. In that example, the interactive window 110 of the first sizedefines a default margin 114 extending outside of the bounding area 116of the first pattern 111 (see FIG. 5A) regardless of the size of thatbounding area. The bounding area 116 of the first pattern 111 is thearea that includes or defines the boundaries of the user's strokes orgestures on the display 102. Therefore, in some examples, the initialsize of the interactive window 110 (i.e., the default margin 114) islarger than the bounding area 116. As described in additional detailbelow, the control unit 320 may adjust the size of the interactivewindow 110 based on the actions of the user.

In another example, the interactive window 110 includes asemi-transparent background (not shown) that may include a distinctivecolor or shading. That semi-transparent background allows the user tosee the images shown on the display 102 through the interactive window110, when the interactive window overlaps with other content displayedon the display. At the same time, the semi-transparent background of theinteractive window 110 provides contrast between a complicatedbackground of the display 102 (e.g., the background of the remote usershown on the display) and the patterns generated on the display 102.

Next, in step 360 of the method 300, the control unit 320 automaticallyexpands the interactive window 110 to a second size in response to asecond pattern 111A on the display 102 to enclose the first pattern 111and the second pattern 111A in the interactive window 110. Very often,users of the display-camera system 200 edit or update the interactive oradded content 225 on the display 102. The method 300 assumes that theuser always generates a second pattern some time after the firstpattern. As shown in FIG. 5B, the user creates a second pattern 111Asome period of time after the first pattern 111 is created. The controlunit 320 determines whether the second pattern 111A is within apredetermined time interval from the first pattern 111 and whether thesecond pattern 111A is at least outside of the bounding area 116 of thefirst pattern 111 to automatically expand the interactive window 110 tothe second size. The predetermined time interval for entry of the secondpattern 111A may be, for example, 30 seconds, 45, seconds, or any otherperiod of time that is chosen by the user and/or stored in the memory310.

When the control unit 320 detects that the second pattern 111A is atleast outside of the bounding area 116 of the first pattern 111 and thesecond pattern 111A is within the predetermined time interval, thecontrol unit 320 automatically encloses the first pattern 111 and thesecond pattern 111A in the expanded interactive window 110 of the secondsize. If the second pattern 111A is within the bounding area 116 of thefirst pattern 111 and is also within the predetermined time frame, thecontrol unit 320 does not expand the interactive window 110. Instead,the control unit 320 includes the second pattern 111A in the interactivewindow 110 of the first size (i.e., the original interactive window).

Further, if the second pattern 111A is outside the bounding area 116 ofthe first pattern but is within the default margin 114 of theinteractive window of the first size, and the second pattern 111A iswithin the predetermined time frame, the control unit 320 automaticallyincludes the second pattern 111A within the interactive window 110 ofthe first size without expanding it. Alternatively, when the secondpattern 111A is within the default margin 114 of the interactive windowof the first size but covers at least a portion of an additional content(e.g., media content) that extends outside of the default margin 114,the control unit can expand the interactive window 110 to a larger sizeand includes the second pattern 111A and the entire media content. Inany event, the interactive window 110 and the content on the window areautomatically saved in the memory 310 by the control unit 320 regardlesswhether the window 110 is expended or not.

FIGS. 6 and 7 are flow charts illustrating examples of alternativemethods for generating and modifying an interactive window. The methodscan be executed by the control unit 320 of the processor 302. Varioussteps described herein with respect to the methods are capable of beingexecuted simultaneously, in parallel, or in an order that differs fromthe illustrated serial manner of execution. The methods are also capableof being executed using additional or fewer steps than are shown in theillustrated example. The methods may be executed in the form ofinstructions encoded in the interaction window module 318 executable bythe processor 302.

The method 400 begins in step 405, where the control unit 320 detects afirst action by a user. In one example, the first action by the userincludes strokes on the display 102 that generate a physical pattern ofadded or interactive content. In another example, the first action bythe user includes user's gestures that generate a virtual pattern ofadded or interactive content (e.g., see the pattern 111 in FIG. 5A).Next, in step 410, the control unit 320 automatically generates anddisplays on the display 102 an interactive window or whiteboard 110 of adefault size in response to the first action by the user. The whiteboard110 of default size defines a margin 114 that is greater than thebounding area 116 (i.e., the area that includes the user's patterns inFIG. 5A) defined by the first action by the user. In one example, thedefault size of the initially created whiteboard 110 may be the sameregardless of the first action or pattern. In another example, thedefault size of the initially created whiteboard 110 may vary dependingon the size of the first action or pattern, where the margin 114 isalways greater than the bounding area 116 of the pattern by somepredetermined amount.

When the first action or pattern covers at least a portion of anyadditional content (e.g., media, etc.) on the display 102, thewhiteboard 110 of default size includes (i.e., incorporates) theadditional content displayed on the display that is at least partiallycovered by the first action. For example, if the display 102 displays animage and during the first action (i.e., pattern) a user covers (i.e.,inks or gestures) at least a portion of that image, the control unit 320includes the entire image in the generated whiteboard 110 of defaultsize. Therefore, the whiteboard 110 of default size is adjustable basedon the action or patter created by the user and the size of theadditional content covered by the pattern. However, in one example, themargin defined by the whiteboard of default size is always greater thanthe bounding area defined by the first action.

In step 415, the control unit 320 determines whether there is a secondaction performed by the user within a first predetermined time intervalor threshold following the first action. In one example, the secondaction includes a physical or a virtual pattern generated by the user.The first predetermined time interval can vary and can be any desiredtime interval. If the control unit 320 determines that there is nosecond action on the display by the user within the predetermined timeinterval, the control unit 320 automatically reduces the size of theinitial whiteboard 110 (at step 417). FIG. 5C illustrates a whiteboard110 that is reduced in size as compared to the initial whiteboard 110 ofFIG. 5A when there is no second action by the user within thepredetermined time period. As shown in FIG. 5C, in one example, thereduced size of the whiteboard 110 is defined by the bounding area 116of the first action (i.e., first pattern) by the user. In other words,if the control unit 320 detects inaction from the user, the control unitsimply adjusts the size of the margin 114 of the whiteboard 110 tocorrespond to the size of the initially added content and saves thatwhiteboard and its content to the memory. Alternatively, the user maymanually reduce the size of the whiteboard 110 at any time.

If, on the other hand, the control unit determines that there is asecond action (i.e., second pattern) by the user, the control unit 320determines an interactive window function F based on the user's action(at step 420). Finally, in step 425, the control unit performs an actionrelated to the interactive window 110 based on interactive windowfunction F determined in step 420.

FIG. 7 illustrates an example of a method 500 for determining aninteractive window function F for the method 400. In step 505, thecontrol unit 320 determines whether the second action by the user iswithin a second predetermined time interval from the first action. Inone example, the second predetermined time interval may the same as thefirst predetermined time interval from step 415. In other examples, thesecond predetermined time interval may be different from the firstpredetermined time interval. If the second action is not within thesecond predetermined time interval from the first action, the controlunit 320 generates and displays a new interactive window or a whiteboard110 based on the second action (at step 510).

If the second action is within the predetermined time interval from thefirst action, the control unit 320 determines whether the second action(i.e., pattern) is within the bounding area 116 of the already existinginteractive window of default size 110 (at step 515). When the secondaction is within the bounding area 116 of the interactive window ofdefault size 110, the control unit includes the second pattern in thealready existing interactive window 110 (at step 520). If the secondaction is not within the bounding area 116 of the existing window 110,the control unit 320 determines whether the second action is within thedefault margin 114 defined by the interactive window of default size 110(at step 522). When the second action is within the default margin 114defined by the existing interactive window of default size 110, thecontrol unit includes the second pattern in the already existinginteractive window 110 (at step 524). Alternatively, when the secondaction is within the default margin 114 defined by the existinginteractive window of default size 110 but the second action covers atleast a portion of additional content (e.g., media content) that extendsoutside of the margin 114, the control unit 320 can extend theinteractive window of default size to a larger size and can include theentire additional content that is at least partially covered by thesecond action.

In an example when the second action is not within the secondpredetermined time interval and is outside of the interactive window ofdefault size (i.e., outside both the bounding area and the defaultmargin), the control unit 320 creates a new interactive window 110.However, a user may wish to include add new content (i.e. a new pattern)to an existing whiteboard after the first or the second predeterminedtime intervals have passed. Therefore, in the same example, if a usermakes an initial stroke or gesture in an already existing whiteboard andthen creates another pattern (outside of the already existingwhiteboard), the control unit 320 can automatically extend the alreadyexisting whiteboard to enclose the new pattern.

Next, when the second action or pattern is not within the default margin114 defined by the existing interactive window, the control unit 320determines whether the second action covers at least a portion ofadditional content or input on the display 102 (i.e., not generated byuser's inking or gesturing) (at step 525). As mentioned above, the addedcontent 225 can include media input (e.g., photos, videos, files, etc.)or other content that can be utilized by the users 104 and 106 duringtheir communication.

FIGS. 8A-B are schematic illustrations showing examples of generatingand modifying an interactive window 110 that includes media content. Asshown in FIG. 8A, the display 102 displays the remote user 106, theoriginal interactive window 110, and the media content 113.

In one example, if the control unit 320 determines that the secondaction or pattern does not cover any additional content (e.g., media),the control unit automatically modifies (e.g., expands) the whiteboardto a larger size (i.e., creates a whiteboard of a second size) toenclose the second action or pattern without including any additionalcontent (at step 530). When the control unit 320 determines that thesecond action or pattern covers at least a portion of any additionalcontent (e.g., media), the control unit automatically expands thewhiteboard to a larger size to include the second pattern and the entireadditional content (e.g., media). In the example shown in FIG. 8B,during the second action or pattern 111A, the user covers (i.e., inks orgestures) a portion of the media content 113 before the secondpredetermined time interval expires. Because the media content 113 isoutside of the bounding area 116 and outside of the default margin 114of the whiteboard 100 of default size (FIG. 8A), the control panel 320automatically expands whiteboard 110 to a larger size to include thesecond action or pattern 112 and the entire media content 113.

In some examples, a user can manipulate any of the created interactivewindows 110 manually. For example, a user can move, collapse, extend, orperform any other manipulations to the interactive windows 110. A usermay also save the interactive windows manually. When the display 102includes two interactive windows 110 (not shown), the control unit 320can merge the two interactive windows when a user clicks on both windowsor drags one window into the other. Alternatively, the control unit 320can capture one interactive window as an image into the otherinteractive window.

What is claimed is:
 1. A method performed by a system, the methodcomprising: automatically generating an interactive window on at least aportion of a display, the interactive window having a first sizecorresponding to a first pattern on the display; and automaticallyexpanding the interactive window to a second size in response to asecond pattern on the display to enclose the first pattern and thesecond pattern in the interactive window; wherein the interactive windowof the first size defines a default margin extending outside of abounding area of the first pattern.
 2. The method of claim 1, whereinthe second pattern is within a predetermined time interval from thefirst pattern and the second pattern is at least outside of the boundingarea of the first pattern to automatically expand the interactive windowto the second size.
 3. The method of claim 1, wherein automaticallyexpanding the interactive window to the second size further comprisesincluding on the interactive window of the second size a contentdisplayed on the display and positioned at least outside of the boundingarea of the first pattern when at least a portion of the content iscovered during the second pattern.
 4. The method of claim 2, furthercomprising automatically reducing the interactive window to a third sizewhen there is no second pattern within the predetermined time intervalfrom the first pattern.
 5. The method of claim 4, wherein the third sizeof the interactive window is defined by the bounding area of the firstpattern.
 6. A system comprising: a computing device connected to animage rendering device, the computing device having a control unit tocause the image rendering device to automatically display on a displayan interactive window of a default size in response to a first action bya user, automatically modify the interactive window to a larger size inresponse to a second action by a user, when the second action isperformed within a predetermined time threshold from the first action.7. The system of claim 6, wherein the first action and the second actioninclude strokes on the display.
 8. The system of claim 7, wherein theinteractive window of default size includes a first content displayed onthe display when the first action covers at least a portion of the firstcontent.
 9. The system of claim 8, wherein the interactive window ofdefault size defines a margin that is greater than a bounding areadefined by the first action, and wherein the control unit is to causethe image rendering device to automatically modify the interactivewindow to a larger size in response to the second action by a user whenthe second action is at least outside of the bounding area.
 10. Thesystem of claim 7, wherein the control unit is to cause the imagerendering device to automatically reduce the size of the interactivewindow when there is no action on the display within the predeterminedtime threshold.
 11. The system of claim 7, wherein the control unit isto cause the image rendering device to display a new interactive windowin response to the second action by a user when the second action isperformed after the predetermined time threshold.
 12. The system ofclaim 7, wherein the interactive window of larger size includes a secondcontent displayed on the display when the second action covers at leasta portion of the second content.
 13. A non-transitory machine-readablestorage medium encoded with instructions executable by a processor of asystem, the machine-readable storage medium comprising instructions to:automatically display on at least a portion of a display an interactivewindow of a default size covering a first pattern by a user, where theinteractive window defines a margin that is greater than a bounding areadefined by the first pattern; and automatically expand the interactivewindow to a larger size in response to a second pattern by a user, whenthe second pattern is within a predetermined time interval from thefirst pattern and the second pattern is at least outside of the boundingarea, wherein the interactive window of default size includes a firstcontent displayed on the display and the first pattern covers least aportion of the first content.
 14. The non-transitory machine-readablestorage medium of claim 13, further comprising instructions to display anew interactive window in response to the second pattern by a user, whenthe second pattern is after the predetermined time interval.
 15. Thenon-transitory machine-readable storage medium of claim 13, furthercomprising instructions to automatically reduce the interactive windowto a smaller size when the second pattern is after the predeterminedtime interval.
 16. The non-transitory machine-readable storage medium ofclaim 13, further comprising instructions to include a second contentwithin the interactive window of larger size when the second patterncovers a least a portion of the second content.